Information processing program, information processing system, information processing apparatus, and information processing method, utilizing augmented reality technique

ABSTRACT

An exemplary embodiment provides an information processing program. The information processing program includes image obtaining instructions, search target detection instructions, event occurrence instructions, virtual image generation instructions, and display control instructions. The search target detection instructions cause a computer to detect a search target from an image of a subject. The event occurrence instructions cause the computer to cause an event to occur in a virtual space in accordance with variation in number of search targets. The virtual image generation instructions cause the computer to generate a virtual image by shooting the event with a virtual camera arranged in the virtual space. The display control instructions cause a display to display the virtual image such that the virtual image is visually recognized by a user as superimposed on the image of the subject or on the subject seen through a screen of the display.

This nonprovisional application is based on Japanese Patent ApplicationNo. 2011-189097 filed with the Japan Patent Office on Aug. 31, 2011, theentire contents of which are hereby incorporated by reference.

FIELD

The invention generally relates to an information processing program, aninformation processing system, an information processing apparatus, andan information processing method, utilizing an augmented realitytechnique.

BACKGROUND AND SUMMARY

An image pick-up apparatus such as a camera has conventionally employeda technique for detecting a prescribed image pick-up target from apicked-up image of a subject (picked-up image). For example, in anaugmented reality technique, a technique for detecting a marker bysubjecting a marker (a search target) included in an image picked up bya camera to image recognition processing (a marker detection technique)has been known.

An exemplary embodiment provides an information processing program, aninformation processing system, an information processing apparatus, andan information processing method, capable of providing more visualeffects for having a user actually feel augmented reality by making useof a detected marker (a search target).

An exemplary embodiment provides a non-transitory storage medium encodedwith a computer readable information processing program and executableby a computer of an information processing apparatus. The informationprocessing apparatus is connected to an image pick-up apparatus and adisplay. The information processing program includes image obtaininginstructions, search target detection instructions, event occurrenceinstructions, virtual image generation instructions, and display controlinstructions. The image obtaining instructions cause the computer toobtain an image of a subject picked up by the image pick-up apparatus.The search target detection instructions cause the computer to detect asearch target from the image of the subject obtained in accordance withthe image obtaining instructions. The event occurrence instructionscause the computer to cause an event to occur in a virtual space inaccordance with variation in number of the search targets that can bedetected in accordance with the search target detection instructions.The virtual image generation instructions cause the computer to generatea virtual image by shooting the event that occurs in accordance with theevent occurrence instructions with a virtual camera arranged in thevirtual space. The display control instructions cause the display todisplay the virtual image such that it is visually recognized by a useras superimposed on the image of the subject or on the subject seenthrough a screen of the display.

According to the exemplary embodiment, since an event is caused to occurin the virtual space in accordance with variation in number of searchtargets that can be detected, more visual effects for having the useractually feel augmented reality by utilizing the search target can beprovided.

In an exemplary embodiment, the search target includes a first searchtarget and a second search target arranged as encompassed in the firstsearch target, and the event occurrence instructions are adapted tocause the event to occur in accordance with the number of the firstsearch targets and the second search targets that can be detected inaccordance with the search target detection instructions.

According to the exemplary embodiment, by detecting the second searchtarget arranged as encompassed in the first search target, more visualeffects for having the user actually feel augmented reality by furtherutilizing the search target can be provided.

In an exemplary embodiment, the event occurrence instructions areadapted to cause a first event to occur when the first search target andthe second search target is successfully detected in accordance with thesearch target detection instructions, and to cause a second event tooccur when only the second search target is successfully detected inaccordance with the search target detection instructions, subsequent tooccurrence of the first event.

According to the exemplary embodiment, since different events are causedto occur in accordance with variation in number of detected searchtargets, more visual effects for having the user actually feel augmentedreality can be provided. For example, the first event may be caused tooccur when both of the first search target and the second search targetis successfully detected, and thereafter, second event may be caused tooccur when the first search target is not detected but only the secondsearch target is successfully detected, for example, by arranging anobject in a real world over the first search target. The first event maybe caused to occur when both of the first search target and the secondsearch target is successfully detected, and thereafter the second eventmay be caused to occur when the first search target is not detected butonly the second search target is successfully detected by bringing theimage pick-up apparatus closer to the second search target. By thusgenerating and displaying a virtual image of the event, more visualeffects for having the user actually feel augmented reality in variousmanners can be provided.

In an exemplary embodiment, the display control instructions are adaptedto display a guide at a position corresponding to the first searchtarget detected in accordance with the search target detectioninstructions.

According to the exemplary embodiment, since the guide is displayed at aposition corresponding to the first search target, the user can performan operation on the search target without confusion and consequently anevent can be caused to occur.

In an exemplary embodiment, the display control instructions are adaptedto display the guide urging the user to take such an action as varyingthe number of the first search targets and the second search targetsthat can be detected in accordance with the search target detectioninstructions.

According to the exemplary embodiment, since the guide urging the userto take such an action as varying the number of the first search targetsand the second search targets is displayed, a hint about causing anevent to occur can be given to the user.

In an exemplary embodiment, the event that occurred in accordance withthe event occurrence instructions includes arrangement of an object in aprescribed shape in the virtual space, and the virtual image generationinstructions are adapted to generate the virtual image by shooting theobject in the prescribed shape with the virtual camera.

According to the exemplary embodiment, since an object in a prescribedshape is displayed in accordance with an event, as compared with a casewhere an object is simply displayed at a position corresponding to thesearch target, more visual effects for having the user actually feelaugmented reality can be provided.

In an exemplary embodiment, the event that occurred in accordance withthe event occurrence instructions includes arrangement of a plate-shapedpolygon in the virtual space, and the virtual image generationinstructions are adapted to generate the virtual image by mapping atexture on the plate-shaped polygon and shooting the polygon with thevirtual camera.

According to the exemplary embodiment, since a virtual image obtained bymapping a texture on the plate-shaped polygon arranged in the virtualspace is generated, more visual effects for having the user actuallyfeel augmented reality can be provided.

In an exemplary embodiment, the information processing program furtherincludes position calculation instructions for calculating positionalrelation between the image pick-up apparatus and the search target basedon the image of the subject and virtual camera arrangement instructionsfor arranging the virtual camera in accordance with the positionalrelation calculated in accordance with the position calculationinstructions.

According to the exemplary embodiment, since an image picked up by avirtual camera arranged in accordance with positional relation betweenthe image pick-up apparatus and the search target is generated as thevirtual image, such display as if an event occurred in a real world canbe provided and hence more visual effects for having the user actuallyfeel augmented reality can be provided.

In an exemplary embodiment, the information processing program furtherincludes first search target position detection instructions fordetecting a position of the first search target from the image of thesubject, first search target position calculation instructions forcalculating positional relation between the image pick-up apparatus andthe first search target based on the position detected in accordancewith the first search target position detection instructions, andvirtual camera arrangement instructions for arranging the virtual camerain accordance with the positional relation calculated in accordance withthe first search target position calculation instructions, and thevirtual image generation instructions are adapted to generate thevirtual image by shooting with the virtual camera, at least the secondevent that occurs in accordance with the event occurrence instructions.

According to the exemplary embodiment, a virtual image is generated byshooting the second event with the virtual camera arranged in accordancewith positional relation calculated in accordance with the first searchtarget position calculation instructions, and therefore more visualeffects for having the user actually feel augmented reality can beprovided. In addition, since the virtual camera is arranged based onpositional relation between the first search target larger than thesecond search target and the image pick-up apparatus, as compared with acase where the virtual camera is arranged based on positional relationbetween the second search target and the image pick-up apparatus, shakeor flicker of the event displayed as the virtual image is lessened.

An exemplary embodiment provides an information processing apparatusconnected to an image pick-up apparatus and a display. The informationprocessing apparatus includes an image obtaining unit, a search targetdetection unit, an event occurrence unit, a virtual image generationunit, and a display control unit. The image obtaining unit obtains animage of a subject picked up by the image pick-up apparatus. The searchtarget detection unit detects a search target from the image of thesubject obtained by the image obtaining unit. The event occurrence unitcauses an event to occur in a virtual space in accordance with variationin number of the search targets that can be detected by the searchtarget detection unit. The virtual image generation unit generates avirtual image by shooting the event that occurs by means of the eventoccurrence unit with a virtual camera arranged in the virtual space. Thedisplay control unit causes the display to display the virtual imagesuch that it is visually recognized by a user as superimposed on theimage of the subject or on the subject seen through a screen of thedisplay.

According to the exemplary embodiment, since an event is caused to occurin the virtual space in accordance with variation in number of searchtargets that can be detected, more visual effects for having the useractually feel augmented reality by utilizing the search target can beprovided.

An exemplary embodiment provides an information processing methodprocessed by a control unit of an information processing apparatusconnected to an image pick-up apparatus and a display. The informationprocessing method includes the steps of obtaining an image of a subjectpicked up by the image pick-up apparatus, detecting a search target fromthe obtained image of the subject, causing an event to occur in avirtual space in accordance with variation in number of the searchtargets that can be detected, generating a virtual image by shooting theevent that occurs with a virtual camera arranged in the virtual space,and causing the display to display the virtual image such that it isvisually recognized by a user as superimposed on the image of thesubject or on the subject seen through a screen of the display.

According to the exemplary embodiment, since an event is caused to occurin the virtual space in accordance with variation in number of searchtargets that can be detected, more visual effects for having the useractually feel augmented reality by utilizing the search target can beprovided.

An exemplary embodiment provides an information processing system. Theinformation processing system includes a search target arranged in asubject, an image pick-up apparatus for picking up an image of thesubject, an information processing apparatus connected to the imagepick-up apparatus, and a display connected to the information processingapparatus. The information processing apparatus includes an imageobtaining unit for obtaining the image of the subject picked up by theimage pick-up apparatus, a search target detection unit for detectingthe search target from the image of the subject obtained by the imageobtaining unit, an event occurrence unit for causing an event to occurin a virtual space in accordance with variation in number of the searchtargets that can be detected by the search target detection unit, avirtual image generation unit for generating a virtual image by shootingthe event that occurs by means of the event occurrence unit with avirtual camera arranged in the virtual space, and a display control unitfor causing the display to display the virtual image such that it isvisually recognized by a user as superimposed on the image of thesubject or on the subject seen through a screen of the display.

According to the exemplary embodiment, since an event is caused to occurin the virtual space in accordance with variation in number of searchtargets that can be detected, more visual effects for having the useractually feel augmented reality by utilizing the search target can beprovided.

The foregoing and other objects, features, aspects and advantages of thepresent invention will become more apparent from the following detaileddescription of the present invention when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary illustrative non-limiting front view of anexemplary non-limiting game device in an opened state according to anexemplary embodiment.

FIG. 2 shows an exemplary illustrative non-limiting left side view ofthe exemplary non-limiting game device in a closed state according tothe exemplary embodiment.

FIG. 3 shows an exemplary illustrative non-limiting top view of theexemplary non-limiting game device in the closed state according to theexemplary embodiment.

FIG. 4 shows an exemplary illustrative non-limiting right side view ofthe exemplary non-limiting game device in the closed state according tothe exemplary embodiment.

FIG. 5 shows an exemplary illustrative non-limiting bottom view of theexemplary non-limiting game device in the closed state according to theexemplary embodiment.

FIG. 6 shows an exemplary illustrative non-limiting block diagramshowing an internal structure of the exemplary non-limiting game deviceaccording to the exemplary embodiment.

FIGS. 7 to 12 each show an exemplary illustrative non-limiting gameimage of the exemplary non-limiting game device according to theexemplary embodiment while a game is being played.

FIG. 13 shows an exemplary illustrative non-limiting schematic diagramshowing a memory map of the exemplary non-limiting game device accordingto the exemplary embodiment.

FIG. 14 shows an exemplary illustrative non-limiting flowchart forillustrating an operation of the exemplary non-limiting game deviceaccording to the exemplary embodiment.

FIG. 15 shows an exemplary illustrative non-limiting flowchart forillustrating processing of a first event shown in FIG. 14.

FIG. 16 shows an exemplary illustrative non-limiting flowchart forillustrating processing of a second event shown in FIG. 14.

FIG. 17 shows an exemplary illustrative non-limiting flowchart forillustrating processing of a third event shown in FIG. 14.

FIGS. 18 and 19 each show an exemplary illustrative non-limiting gameimage of the exemplary non-limiting game device according to anotherexemplary embodiment while the game is being played.

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS

(Configuration of Game Device)

A game device according to one embodiment will be described withreference to FIGS. 1 to 5. A game device 10 is a portable game device.

Game device 10 has a lower housing 11 and an upper housing 21. Lowerhousing 11 and upper housing 21 are connected to each other to allowopening and closing (to be foldable).

(Description of Lower Housing)

Lower housing 11 is provided with a lower LCD (Liquid Crystal Display)12, a touch panel 13, operation buttons 14A to 14L, an analog stick 15,LEDs 16A to 16B, an insertion port 17, and a microphone hole 18.

Touch panel 13 is attached onto a screen of lower LCD 12. In an uppersurface of lower housing 11, insertion port 17 (a dotted line shown inFIGS. 1 and 5) for accommodating a touch pen 28 is provided.

On an inner surface (main surface) of lower housing 11, a cross-shapedbutton (a direction input button) 14A, a button 14B, a button 14C, abutton 14D, a button 14E, a power button 14F, a select button 141, aHOME button 14K, and a start button 14L are provided.

Analog stick 15 is a device indicating a direction.

In the inner surface of lower housing 11, microphone hole 18 isprovided. Under microphone hole 18, a microphone 42 (see FIG. 6) servingas an audio input apparatus is provided.

As shown in FIGS. 3 and 5, an L button 14G and an R button 14H areprovided on the upper surface of lower housing 11. In addition, as shownin FIG. 2, on a left side surface of lower housing 11, a sound volumebutton 141 for adjusting volume of a speaker 43 (see FIG. 6) included ingame device 10 is provided.

As shown in FIG. 2, a cover portion 11C that can be opened and closed isprovided on the left side surface of lower housing 11. In the inside ofthis cover portion 11C, a connector for electrically connecting gamedevice 10 and an external memory for data saving 45 to each other isprovided.

As shown in FIG. 5, an insertion port 11D for inserting an externalmemory 44 is provided in the upper surface of lower housing 11.

As shown in FIGS. 1 and 4, a first LED 16A for notifying a user of apower ON/OFF state of game device 10 is provided on a lower surface oflower housing 11 and a second LED 16B for notifying the user of a stateof establishment of wireless communication of game device 10 is providedon a right side surface of lower housing 11. Game device 10 is capableof establishing wireless communication with other equipment and awireless switch 19 for activating/inactivating this wirelesscommunication function is provided on the right side surface of lowerhousing 11 (see FIG. 4).

(Description of Upper Housing)

As shown in FIGS. 1 and 2, upper housing 21 is provided with an upperLCD 22, an outer image pick-up portion 23 (an outer image pick-upportion (left) 23 a and an outer image pick-up portion (right) 23 b), aninner image pick-up portion 24, a 3D adjustment switch 25, and a 3Dindicator 26.

Upper LCD 22 is a display capable of displaying an image that canstereoscopically be viewed. Specifically, upper LCD 22 is a parallaxbarrier type display allowing stereoscopic vision with naked eyes. UpperLCD 22 can display an image with stereoscopic effect as felt by a user(a stereoscopic image (3D image)) by having the users left eye visuallyrecognize an image for left eye and the user's right eye visuallyrecognize an image for right eye by using a parallax barrier. Inaddition, upper LCD 22 can inactivate the parallax barrier above, andwhen the parallax barrier is inactivated, an image can two-dimensionallybe displayed. Thus, upper LCD 22 is a display capable of switchingbetween a stereoscopic display mode for displaying a stereoscopic imageand a two-dimensional display mode for two-dimensionally displaying animage (displaying a two-dimensional image). Switching between thedisplay modes is made, for example, by 3D adjustment switch 25 whichwill be described later.

Outer image pick-up portion 23 is a generic name of two image pick-upportions (23 a and 23 b) provided on an outer surface 21D of upperhousing 21. Outer image pick-up portion (left) 23 a and outer imagepick-up portion (right) 23 b can be used as stereo cameras by a programexecuted by game device 10.

Inner image pick-up portion 24 is an image pick-up portion provided onan inner surface 21B of upper housing 21 and having an inward directionof normal to the inner surface as a direction of image pick-up.

Three-D adjustment switch 25 is a slide switch and it is a switch usedfor switching between the display modes of upper LCD 22 as describedabove. In addition, 3D adjustment switch 25 is used for adjustingstereoscopic effect of an image that is displayed on upper LCD 22 andcan stereoscopically be viewed (a stereoscopic image). A slider 25 a of3D adjustment switch 25 can be slid to any position in a prescribeddirection (a vertical direction) and a display mode of upper LCD 22 isset in accordance with a position of slider 25 a. Further, how astereoscopic image is viewed is adjusted in accordance with a positionof slider 25 a.

Three-D indicator 26 is an LED indicating whether upper LCD 22 is in astereoscopic display mode or not.

Further, a speaker hole 21E is provided in the inner surface of upperhousing 21. Voice and sound from speaker 43 which will be describedlater is output from this speaker hole 21E.

(Internal Configuration of Game Device 10)

An internal electrical configuration of game device 10 will be describedwith reference to FIG. 6. As shown in FIG. 6, game device 10 includes,in addition to each portion described above, such electronic componentsas an information processing unit 31, a main memory 32, an externalmemory interface (external memory I/F) 33, an external memory I/F fordata saving 34, an internal memory for data saving 35, a wirelesscommunication module 36, a local communication module 37, a real timeclock (RTC) 38, an acceleration sensor 39, a power supply circuit 40,and an interface circuit (I/F circuit) 41.

Information processing unit 31 includes a CPU (Central Processing Unit)311 for executing a prescribed program, a GPU (Graphics Processing Unit)312 for performing image processing, and a VRAM (Video RAM) 313. CPU 311performs processing in accordance with a program by executing theprogram stored in a memory in game device 10 (such as external memory 44connected to external memory I/F 33 or internal memory for data saving35). It is noted that a program executed by CPU 311 may be obtained fromother equipment through communication with other equipment. GPU 312generates an image in response to an instruction from CPU 311 andrenders the image in VRAM 313. The image rendered in VRAM 313 is outputto upper LCD 22 and/or lower LCD 12 so that the image is displayed onupper LCD 22 and/or lower LCD 12.

External memory I/F 33 is an interface for removably connecting externalmemory 44. In addition, external memory I/F for data saving 34 is aninterface for removably connecting external memory for data saving 45.

Main memory 32 is a volatile storage device used as a work area or abuffer area of (CPU 311 of) information processing unit 31.

External memory 44 is a non-volatile storage device for storing aprogram and the like executed by information processing unit 31.External memory 44 is implemented, for example, by a read-onlysemiconductor memory.

External memory for data saving 45 is implemented by a non-volatilereadable and writable memory (such as a NAND type flash memory) and usedfor saving any data.

Internal memory for data saving 35 is implemented by a non-volatilereadable and writable memory (such as a NAND type flash memory) and usedfor storing prescribed data. For example, internal memory for datasaving 35 stores data or a program downloaded through wirelesscommunication via wireless communication module 36.

Wireless communication module 36 has a function to establish connectionto wireless LAN under a scheme complying, for example, withIEEE802.11b/g standard, Local communication module 37 has a function toestablish wireless communication with a game device of the same typeunder a prescribed communication scheme (such as communication under anoriginal protocol or infrared communication).

Acceleration sensor 39 detects magnitude of acceleration in a directionof a straight line (linear acceleration) along each direction of threeaxes (xyz axes). Information processing unit 31 can detect anorientation or motion of game device 10 as it receives data indicatingacceleration (acceleration data) detected by acceleration sensor 39.

RTC 38 counts time and outputs the time to information processing unit31. Information processing unit 31 calculates the current time (date)based on the time counted by RTC 38, Power supply circuit 40 controlselectric power from a power supply (a chargeable battery) included ingame device 10 and causes supply of electric power to each component ingame device 10.

Touch panel 13, microphone 42 and speaker 43 are connected to I/Fcircuit 41. I/F circuit 41 includes an audio control circuit forcontrolling microphone 42 and speaker 43 (an amplifier) and a touchpanel control circuit for controlling the touch panel. The audio controlcircuit carries out A/D conversion and D/A conversion of an audio signaland conversion of an audio signal to audio data in a prescribed format.The touch panel control circuit generates touch position data in aprescribed format based on a signal from touch panel 13 and outputs thetouch position data to information processing unit 31. As informationprocessing unit 31 obtains the touch position data, it can know aposition where input onto touch panel 13 has been made.

Operation button 14 consists of operation buttons 14A to 14L above andoperation data indicating a status of input to operation buttons 14A to14I (whether an operation button has been pressed or not) is output fromoperation button 14 to information processing unit 31.

Lower LCD 12 and upper LCD 22 are connected to information processingunit 31. Specifically, information processing unit 31 is connected anLCD controller (not shown) of upper LCD 22 and controls the LCDcontroller to turn ON/OFF the parallax barrier. When the parallaxbarrier of upper LCD 22 is turned ON, an image for right eye and animage for left eye stored in VRAM 313 of information processing unit 31are output to upper LCD 22. More specifically, the LCD controller readsthe image for right eye and the image for left eye from VRAM 313 byalternately repeating processing for reading pixel data for one line ina vertical direction with regard to the image for right eye andprocessing for reading pixel data for one line in the vertical directionwith regard to the image for left eye. Thus, the image for right eye andthe image for left eye are each divided into strip-like images in whichpixels are aligned vertically for each one line, and an image in whichstrip-like images of the image for right eye and strip-like images ofthe image for left eye as divided are alternately arranged is displayedon a screen of upper LCD 22. Then, as the image is visually recognizedby the user through the parallax barrier of upper LCD 22, the user'sright and left eyes visually recognize the image for right eye and theimage for left eye respectively. As described above, an image that canstereoscopically be viewed is displayed on the screen of upper LCD 22.

Outer image pick-up portion 23 and inner image pick-up portion 24 pickup an image in response to an instruction from information processingunit 31 and output data of the picked-up image to information processingunit 31.

Three-D adjustment switch 25 transmits an electric signal in accordancewith a position of slider 25 a to information processing unit 31.

Information processing unit 31 controls illumination of 3D indicator 26.For example, when upper LCD 22 is in a stereoscopic display mode,information processing unit 31 causes 3D indicator 26 to illuminate.

(Outlines of Game)

Outlines of a game performed in game device 10 will be described withreference to FIGS. 7 to 12. This game is a game making use of anaugmented reality technique and played while an image of a marker 50 (anouter marker 50 a, an inner marker 50 b) in a prescribed design (acharacter, a number, a symbol, other designs) drawn in a note comingtogether with game software is picked up by outer image pick-up portion23 of game device 10.

A game image example shown in FIG. 7 shows a game image displayed onupper LCD 22 and lower LCD 12 when marker 50 is detected immediatelyafter the game is started.

On upper LCD 22, an image of a note (a subject) currently picked up byouter image pick-up portion 23 and an image of marker 50 drawn in thenote are displayed. In this example, marker 50 has outer marker 50 a (afirst search target) and inner marker 50 b arranged as encompassed inouter marker 50 a (a second search target).

On lower LCD 12, information indicating that marker 50 is being detected(for example, “detecting marker”) is displayed. When game device 10failed to detect marker 50, game device 10 may display such informationas “Marker has not been detected. Please perform again processing fordetecting marker.” on lower LCD 12. When game device 10 detects marker50, game device 10 may display such information as “Marker has beendetected.” on lower LCD 12.

When two markers of outer marker 50 a and inner marker 50 b issuccessfully detected, game device 10 causes a first event to occur. Asa result of the first event that occurred, game device 10 reproduces amotion picture in a region within outer marker 50 a of upper LCD 22 andthereafter displays a button object 60 arranged within outer marker 50 aas shown in FIG. 8.

Button object 60 displayed on upper LCD 22 is a guide for having theuser make transition to a second event and it is displayed at a positioncorresponding to outer marker 50 a. It is noted that there is also acase where, simply by displaying button object 60, the user cannotunderstand an action that he/she should take next, and therefore gamedevice 10 may display such information as “Press button.” on lower LCD12 in order to urge the user to take action. Namely, by displaying aguide urging the user to take such an action as varying the number ofmarkers, a hint about causing an event to occur can be given to theuser.

Then, when the user presses button object 60 displayed on upper LCD 22as shown in FIG. 9, game device 10 cannot detect outer marker 50 abecause a part of outer marker 50 a is hidden by a user's hand. It isnoted that game device 10 makes determination as to whether marker 50can be detected or not in the following detection processing.

Game device 10 detects a series of edge pixels from an image of marker50 through edge detection processing and obtains straight lines of foursides based on the series of edge pixels. Here, four vertices arecalculated by detecting an intersection of two adjacent straight linesof four straight lines. By thus calculating four vertices, four verticescan be obtained, for example, even when a part of an outer frame ofouter marker 50 a is hidden by a hand.

Then, game device 10 extracts an image of the inside of outer marker 50a (a region surrounded by the four vertices above), calculatessimilarity by comparing data of the extracted image with pattern datastored in advance in a game program, and determines that outer marker 50a in accordance with the pattern data has been detected when thecalculated similarity is not smaller than a prescribed value.

Then, positional relation between outer image pick-up portion 23 andouter marker 50 a in a real space is calculated from the four vertices,and a position and an orientation of the virtual camera in the virtualspace are calculated based on this result of calculation. The virtualcamera is installed such that positional relation between outer imagepick-up portion 23 and outer marker 50 a in the real space andpositional relation between the virtual camera and the marker in thevirtual space correspond to each other.

Game device 10 extracts such an image of the inside of outer marker 50 aof which part is hidden by a user's hand, calculates similarity bycomparing data of the extracted image with the pattern data stored inadvance in the game program, and determines that outer marker 50 acannot be detected when it is determined that the calculated similarityis smaller than the prescribed value.

It is noted that, even though game device 10 determines that outermarker 50 a cannot be detected, it can obtain four vertices of outermarker 50 a. Therefore, even in a case where only inner marker 50 b hasbeen detected in processing of the second event, if setting of thevirtual camera is based on the four vertices of outer marker 50 a,positions of respective four vertices are stabilized as compared with acase where the virtual camera is set based on four vertices of innermarker 50 b and flicker of a displayed virtual object can be lessened.

In addition, determination by game device 10 as to whether marker 50 canbe detected or not is not limited to the method described previously,and for example, a method of making determination based on whether allsides of marker 50 can be recognized or not may be applicable. Forexample, if a part of outer marker 50 a is hidden by a user's hand andone of the four sides of outer marker 50 a cannot be recognized, gamedevice 10 determines that outer marker 50 a cannot be detected.

When game device 10 can no longer detect outer marker 50 a because theuser presses button object 60 and a part of outer marker 50 a is hiddenby a user's hand, the number of markers 50 that can be detected varies.Namely, when the user presses button object 60 in a state where gamedevice 10 is being able to detect two markers of outer marker 50 a andinner marker 50 b, game device 10 will no longer be able to detect outermarker 50 a and to detect only inner marker 50 b.

When the number of markers 50 that can be detected varies afterprocessing of the first event ends, game device 10 causes the secondevent to occur. As the second event occurs, on upper LCD 22, a characterobject 61 is displayed within inner marker 50 b as shown in FIG. 10. Itis noted that game device 10 may display such information as “Removeyour hand from button.” on lower LCD 12 in order to have the user maketransition to a third event.

Then, when the user removes his/her hand from button object 60 displayedon upper LCD 22 as shown in FIG. 11, game device 10 can again recognizea part of outer marker 50 a that has been hidden by the user's hand andhence it can detect outer marker 50 a.

As the user removes his/her hand from button object 60, game device 10can again recognize a part of outer marker 50 a that has been hidden bythe user's hand and can detect outer marker 50 a, and thus the number ofmarkers 50 that can be detected varies. Namely, when the user removeshis/her hand from button object 60 while game device 10 is being able todetect a marker, i.e., only inner marker 50 b, it will again be able todetect outer marker 50 a and enter a state where it can detect twomarkers of outer marker 50 a and inner marker 50 b.

When the number of markers 50 that can be detected varies afterprocessing of the second event ends, game device 10 causes the thirdevent to occur. As a result of the third event that occurred, on upperLCD 22, a situation in which character object 61 moves in a direction ofthe user's removed hand is displayed as shown in FIG. 12.

(Operation of Game Device 10)

An operation of game device 10 will now be described in further detailwith reference to FIGS. 13 to 17.

Main memory 32 shown in FIG. 13 includes a game program D11, motionpicture information D12, virtual object information D13, a left subjectimage D14, a right subject image D15, virtual camera information D16, aleft virtual space image D17, a right virtual space image D18, and thelike.

Game program D11 is read from external memory 44, internal memory fordata saving 35 or the like and stored in main memory 32 prior to play ofthe game. It is noted that game program D11 may be received from aserver apparatus or another game device by using a wirelesscommunication function and stored in main memory 32.

Motion picture information D12 is motion picture information displayedin the first event or the like, and it is motion picture informationcompressed based on such motion picture compression standards as H.264,MPEG-4 AVC, or the like.

Virtual object information D13 is information relating to variousvirtual objects (button object 60 and character object 61) arranged inthe virtual space and it includes such information as a shape, a color,a design, a position, and an orientation of a virtual object. Regardingcharacter object 61, information on a direction of movement and a movingspeed is also included.

Left subject image D14 is an image of a subject picked up by outer imagepick-up portion (left) 23 a and right subject image D15 is an image of asubject picked up by outer image pick-up portion (right) 23 b.

Virtual camera information D16 is information on a pair of virtualcameras (a left virtual camera and a right virtual camera which will bedescribed later) used in generating a stereoscopic image, and itincludes such information as a position or an orientation of eachvirtual camera in the virtual space.

Left virtual space image D17 is a virtual image rendered based on theleft virtual camera and right virtual space image D18 is a virtual imagerendered based on the right virtual camera.

Flow of game processing performed by CPU 311 based on game program D11will be described with reference to FIG. 14. The game processing isstarted, for example, when button 14B of operation button 14 is pressedwhile an image of a subject is picked up by outer image pick-up portion23.

Initially, CPU 311 obtains most recent left subject image D14 and rightsubject image D15 from outer image pick-up portion (left) 23 a and outerimage pick-up portion (right) 23 b and causes main memory 32 to storethe images (step S11).

Then, CPU 311 detects marker 50 from the obtained image of the subject(step S12). In the marker detection processing, marker 50 in each ofleft subject image D14 and right subject image D15 is detected. Forexample, left subject image D14 and right subject image D15 are eachsubjected to edge detection processing and pattern matching processing,so as to detect marker 50 in each of left subject image D14 and rightsubject image D15. Here, in the edge detection processing, a series ofedge pixels representing a contour of marker 50 is detected from leftsubject image D14 and right subject image D15 and a plurality ofstraight lines are generated based on the series of edge pixels, tothereby detect vertices of the contour.

Then, CPU 311 calculates positional relation between outer image pick-upportion 23 and marker 50 in the real space based on four vertices ofmarker 50 and calculates a position and an orientation of the virtualcamera in the virtual space based on this result of calculation, and thevirtual camera is installed such that positional relation between outerimage pick-up portion 23 and outer marker 50 a in the real space andpositional relation between the virtual camera and the marker in thevirtual space correspond to each other (step S13). Processing forcalculating a position of the virtual camera based on the positionalrelation above may be similar to that in a conventional augmentedreality feeling technique. It is noted that processing for calculating aposition and an orientation of the virtual camera in the virtual spaceand installing the virtual camera is performed at a certain interval andit is performed also during processing of each event which will bedescribed later, In a subsequent flowchart, however, the processing isassumed as being performed at a certain interval including duringprocessing of each event which will be described later, although thedescription may not be provided. In addition, here, similar processingmay be performed also for marker 50 b, a virtual camera separate fromthe virtual camera above may be arranged in the virtual space such thatpositional relation between outer image pick-up portion 23 and innermarker 50 b in the real space and positional relation between thevirtual camera and marker 50 b in the virtual space correspond to eachother, and thus the virtual camera may be used in step S172 which willbe described later.

As shown in FIG. 7, marker 50 has outer marker 50 a and inner marker 50b arranged as encompassed in outer marker 50 a. It is noted that marker50 is not limited to a case having outer marker 50 a and inner marker 50b, and a case where a plurality of inner markers 50 b are encompassed inouter marker 50 a, a case where a marker is further encompassed in innermarker 50 b, and the like may be applicable.

In a case where marker 50 has outer marker 50 a and inner marker 50 b.CPU 311 detects outer marker 50 a by performing edge detectionprocessing and pattern matching processing, and thereafter detects innermarker 50 b by again performing edge detection processing and patternmatching processing. It is noted that CPU 311 may detect outer marker 50a and inner marker 50 b together by once performing edge detectionprocessing and pattern matching processing.

Then, CPU 311 determines whether it has detected two markers of outermarker 50 a and inner marker 50 b or not (step S14). When CPU 311determines that it failed to detect two markers 50 of outer marker 50 aand inner marker 50 b (step S14: NO), the process returns to step S12and processing for detecting marker 50 from the obtained image of thesubject is continued. When CPU 311 determines that it has detected twomarkers of outer marker 50 a and inner marker 50 b (step S14: YES), CPU311 causes the first event to occur (step S15).

Processing of the first event that occurred will now be described withreference to FIG. 15. Initially, CPU 311 reads motion pictureinformation D12 from main memory 32 and reproduces the motion picture ina region within outer marker 50 a (step S151). Specifically, a method ofreproducing the motion picture in the region within outer marker 50 awill be described. CPU 311 arranges a plate-shaped polygon in thevirtual space in response to the first event that occurred. CPU 311generates a virtual image by mapping an image of motion pictureinformation D12 as a texture on the arranged plate-shaped polygon andthen shooting the image with the virtual camera. CPU 311 reproduces themotion picture by successively displaying the virtual images on upperLCD 22 such that the generated virtual images as superimposed on theimage of the subject are visually recognized by the user.

After the motion picture is reproduced, CPU 311 reads button object 60from virtual object information D13 in main memory 32 and arranges readbutton object 60. Thereafter, CPU 311 generates a virtual image withinthe virtual space by picking up an image of the virtual space includingbutton object 60 with the virtual camera, synthesizes the image of thesubject with the virtual image, and displays the resultant image onupper LCD 22 (step S152).

The processing of the first event ends when button object 60 isdisplayed on upper LCD 22. CPU 311 urges the user of an operation topress button object 60 by displaying button object 60 on upper LCD 22.It is noted that CPU 311 is assumed to continually perform processingfor detecting marker 50 after processing of the first event ended.Therefore, as the user presses button object 60, a part of outer marker50 a is hidden by a user's hand and outer marker 50 a can no longer bedetected. Namely, CPU 311 can determine whether the user has pressedbutton object 60 or not by determining whether it has detected onlyinner marker 50 b or not.

Referring back to FIG. 14, CPU 311 determines whether it has detectedonly inner marker 50 b or not (step S16). When CPU 311 determines thatit failed to detect only inner marker 50 b (when CPU 311 determines thatit has detected two markers of outer marker 50 a and inner marker 50 band/or CPU 311 determines that it has detected neither of outer marker50 a and inner marker 50 b) (step S16: NO), CPU 311 continues processingfor detecting marker 50. When CPU 311 determines that it has detectedonly inner marker 50 b (step S16: YES), it causes the second event tooccur (step S17).

Processing of the second event that occurred will now be described withreference to FIG. 16. Initially, CPU 311 arranges character object 61 ata prescribed position with the origin in the virtual space serving asthe reference, the origin being the center of outer marker 50 a (stepS171).

After CPU 311 arranged character object 61, CPU 311 generates a virtualimage in the virtual space by picking up an image of the virtual spaceincluding character object 61 with the virtual camera (step S172).Specifically, CPU 311 generates left virtual space image D17 byrendering a virtual object (such as character object 61) based on theleft virtual camera and generates right virtual space image D18 byrendering a virtual object based on the right virtual camera. It isnoted that processing for rendering the virtual space is typicallyperformed by GPU 312 in response to an instruction from CPU 311.

After CPU 311 generates a virtual image, CPU 311 synthesizes the imageof the subject with the virtual image and causes upper LCD 22 to displaythe resultant image (step S173). Specifically, CPU 311 overwrites leftsubject image D14 with left virtual space image D17 (that is, characterobject 61 rendered based on the left virtual camera) and overwritesright subject image D15 with right virtual space image D18 (that is,character object 61 rendered based on the right virtual camera). Thenthe synthesized image is displayed on upper LCD 22. Namely, CPU 311causes upper LCD 22 to display a 3D image, with the synthesized image ofleft subject image D14 and left virtual space image D17 serving as animage for left eye and the synthesized image of right subject image D15and right virtual space image D18 serving as an image for right eye.

The processing of the second event ends when character object 61 isdisplayed on upper LCD 22. It is noted that CPU 311 is assumed tocontinually perform processing for detecting marker 50 after processingof the second event ended. Therefore, as the user removes his/her handfrom button object 60, a part of outer marker 50 a that has been hiddenby a user's hand is again recognized and outer marker 50 a can again bedetected. Namely, CPU 311 can determine whether the user has removedhis/her hand from button object 60 or not by determining whether it hasdetected two markers of outer marker 50 a and inner marker 50 b or not.

Referring back to FIG. 14, CPU 311 detects whether it has detected twomarkers of outer marker 50 a and inner marker 50 b or not (step S18).When CPU 311 determines that it failed to detect two markers of outermarker 50 a and inner marker 50 b (step S18: NO), CPU 311 continuesprocessing for detecting marker 50. When CPU 311 determines that it hasdetected two markers of outer marker 50 a and inner marker 50 b (stepS18: YES), CPU 311 causes the third event to occur (step S19).

Processing of the third event that occurred will now be described withreference to FIG. 17. Initially, CPU 311 moves arranged character object61 (step S191). Specifically, CPU 311 determines a direction of movementand a moving speed of character object 61 based on virtual objectinformation D13 read from main memory 32. For example, CPU 311determines that character object 61 is to be moved at a slow speed in adirection of button object 60 pressed by the user. Then, when CPU 311moved character object 61 based on the determination, CPU 311 updates aposition of moved character object 61.

CPU 311 generates a virtual image in the virtual space by picking up animage of the virtual space including moving character object 61 with thevirtual camera (step S192). Specifically, CPU 311 generates left virtualspace image D17 by rendering moving character object 61 based on theleft virtual camera and generates right virtual space image D18 byrendering moving character object 61 based on the right virtual camera.

It is noted that, by using the virtual camera arranged in the processingof the second event, it is not necessary to again arrange the virtualcamera. If the virtual camera is to be arranged again, CPU 311 againcalculates positional relation between outer image pick-up portions 23a, 23 b and marker 50 and again arranges a pair of virtual cameras (theleft virtual camera and the right virtual camera) in the virtual spacebased on the positional relation again calculated.

After the virtual image is generated, CPU 311 synthesizes the image ofthe subject with the virtual image and causes upper LCD 22 to displaythe resultant image (step S193). Specifically, CPU 311 successivelyoverwrites left subject image D14 with moving character object 61rendered based on the left virtual camera and successively overwritesright subject image D15 with moving character object 61 rendered basedon the right virtual camera. Thus, the user can be caused to recognizethat character object 61 moves in the virtual space. It is noted that,similarly, the synthesized image is displayed on upper LCD 22. Namely,CPU 311 causes upper LCD 22 to display a 3D image, with the synthesizedimage of left subject image D14 and left virtual space image D17 servingas an image for left eye and the synthesized image of right subjectimage D15 and right virtual space image D18 serving as an image forright eye.

Referring back to FIG. 14, CPU 311 ends the game after processing of thethird event ended. It is noted that the process may return to step S12and CPU 311 may continue processing for detecting marker 50 from theobtained image of the subject, without ending the game.

As described above, game device 10 obtains an image of a subject pickedup by outer image pick-up portion 23, detects marker 50 from theobtained image of the subject, causes an event to occur in the virtualspace in accordance with variation in number of markers 50 that can bedetected, generates a virtual image by shooting the event that occurswith a virtual camera arranged in the virtual space, and causes upperLCD 22 to display the virtual image such that the virtual image isvisually recognized by the user as superimposed on the image of thesubject. Thus, more visual effects for having the user actually feelaugmented reality by making use of marker 50 can be given.

(Variation)

In the embodiment above, a case where outer marker 50 a cannot bedetected because a part of outer marker 50 a is hidden by a user's handand hence the number of markers 50 that can be detected is varied hasbeen described, however, game device 10 is not limited as such. Forexample, as shown in FIGS. 18 and 19, game device 10 may vary the numberof markers 50 that can be detected, by varying the number of markers 50of which images are picked up by outer image pick-up portion 23.

As shown in FIG. 18, game device 10 picks up an image of a subject suchthat the image of the subject currently picked by outer image pick-upportion 23 includes outer marker 50 a and inner marker 50 b. Namely,game device 10 picks up an image of the subject with a distance from thesubject being sufficiently secured. Here, since the image of the subjectincludes outer marker 50 a and inner marker 50 b, game device 10 candetect outer marker 50 a and inner marker 50 b.

Then, the user brings game device 10 closer to the subject. As shown inFIG. 19, when game device 10 is brought closer to the subject, gamedevice 10 picks up an image of the subject such that the image of thesubject currently picked up by outer image pick-up portion 23 includesonly inner marker 50 b. Here, since the image of the subject includesonly inner marker 50 b, game device 10 cannot detect outer marker 50 abut can detect only inner marker 50 b. As the number of markers 50 thatcan be detected varies, game device 10 causes an event to occur. As aresult of the event that occurred, as shown in FIG. 19, upper LCD 22displays character object 61 within inner marker 50 b.

As described above, by bringing game device 10 closer to the subject,the number of markers 50 that can be detected can be varied. It is notedthat, by moving game device 10 away from the subject, inner marker 50 bwill be in such a size as undetectable and hence the number of markers50 that can be detected can also be varied. Alternatively, the number ofmarkers 50 that can be detected can be varied also by moving a part ofouter marker 50 a out of the frame, without changing a distance of gamedevice 10 from the subject.

In addition, in the embodiment above, upper LCD 22 displays asynthesized image of an image of the subject picked up by outer imagepick-up portion 23 and a virtual image (a video see-through type),however, the embodiment is not limited as such. In other embodiments,such an optical see-through type game device (for example, a game deviceincluding a head-mount display) or game system that a virtual object isdisplayed on a transmissive type display screen and the virtual objectis visually recognized by the user as if the virtual object wereactually present in the subject seen through the transmissive typedisplay screen is also applicable.

Moreover, in the embodiment above, though a position and an orientationof the left virtual camera and a position and an orientation of theright virtual camera are individually determined in accordance withresults in the marker detection processing, in other embodiments, aposition and an orientation of any one virtual camera may be determinedin accordance with results in the marker detection processing andthereafter a position and an orientation of the other virtual camera maybe determined based on the position and the orientation of one virtualcamera.

Further, in the embodiment above, though a position and an orientationof the virtual camera are determined based on a position and anorientation of marker 50 included in the image of the subject, in otherembodiments, a position or the like of the virtual camera may bedetermined based on a position or the like of a recognition target (forexample, a face, a hand, and the like) other than marker 50 included inthe image of the subject.

Furthermore, in the embodiment above, though an example where astereoscopic image is displayed on upper LCD 22 has been described, acase where a two-dimensional image is displayed is also applicable,without limited as such. For example, one virtual camera should only bearranged in the virtual space based on the image of the subject obtainedfrom any one of outer image pick-up portion (left) 23 a and outer imagepick-up portion (right) 23 b and then processing as in the embodimentabove should only be performed.

In addition, in the embodiment above, though an example in whichprocessing of game program D11 is performed by the CPU (CPU 311) hasbeen described, in other embodiments, at least partial processing ofgame program D11 may be performed only by hardware.

Moreover, in the embodiment above, though description has been givenwith reference to the game example above, the embodiment is applicableto any such game as causing an event to occur as the number of markers50 that can be detected varies, without limited as such. Any applicationsoftware is also applicable, without limited to game software.

Further, in the embodiment above, though an example where a note wheremarker 50 is drawn is used has been described, a case where marker 50 isdrawn on a common credit card or a card having a size as large as abusiness card is also applicable, without limited as such. Here, a cardin which marker 50 is drawn may simply be referred to as a marker.

Furthermore, in the embodiment above, though portable game device 10 hasbeen described, in other embodiments, any information processingapparatus (a stationary game device, a desktop personal computer, anotebook personal computer, a portable telephone, and the like) may bemade use of as a game device or a game system.

In addition, in the embodiment above, though an example where processingof game program D11 is performed by a single CPU (CPU 311) has beendescribed, in other embodiments, processing of game program D11 may beperformed by a plurality of processors as distributed. The plurality ofprocessors may be contained in the same information processing apparatusor may be provided as distributed in a plurality of informationprocessing apparatuses capable of communicating with one another, suchas a server apparatus and a client apparatus.

While certain example systems, methods, devices, and apparatuses havebeen described herein, it is to be understood that the appended claimsare not to be limited to the systems, methods, devices, and apparatusesdisclosed, but on the contrary, are intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

1. A non-transitory storage medium encoded with a computer readableinformation processing program and executable by a computer of aninformation processing apparatus connected to an image pick-up apparatusand a display, said information processing program comprising: imageobtaining instructions for obtaining an image of a subject picked up bysaid image pick-up apparatus; search target detection instructions fordetecting a search target from the image of said subject obtained inaccordance with said image obtaining instructions; event occurrenceinstructions for causing an event to occur in a virtual space inaccordance with variation in number of said search targets that can bedetected in accordance with said search target detection instructions;virtual image generation instructions for generating a virtual image byshooting said event that occurs in accordance with said event occurrenceinstructions with a virtual camera arranged in said virtual space; anddisplay control instructions for causing said display to display saidvirtual image such that it is visually recognized by a user assuperimposed on the image of said subject or on said subject seenthrough a screen of said display.
 2. The non-transitory storage mediumaccording to claim 1, wherein said search target includes a first searchtarget and a second search target arranged as encompassed in said firstsearch target, and said event occurrence instructions are adapted tocause the event to occur in accordance with the number of said firstsearch targets and said second search targets that can be detected inaccordance with said search target detection instructions.
 3. Thenon-transitory storage medium according to claim 2, wherein said eventoccurrence instructions are adapted to cause a first event to occur whensaid first search target and said second search target is successfullydetected in accordance with said search target detection instructions,and cause a second event to occur when only said second search target issuccessfully detected in accordance with said search target detectioninstructions, subsequent to occurrence of said first event.
 4. Thenon-transitory storage medium according to claim 2, wherein said displaycontrol instructions are adapted to display a guide at a positioncorresponding to said first search target detected in accordance withsaid search target detection instructions.
 5. The non-transitory storagemedium according to claim 4, wherein said display control instructionsare adapted to display said guide urging said user to take such anaction as varying the number of said first search targets and saidsecond search targets that can be detected in accordance with saidsearch target detection instructions.
 6. The non-transitory storagemedium according to claim 1, wherein said event that occurred inaccordance with said event occurrence instructions includes arrangementof an object in a prescribed shape in said virtual space, and saidvirtual image generation instructions are adapted to generate saidvirtual image by shooting the object in said prescribed shape with saidvirtual camera.
 7. The non-transitory storage medium according to claim6, wherein said event that occurred in accordance with said eventoccurrence instructions includes arrangement of a plate-shaped polygonin said virtual space, and said virtual image generation instructionsare adapted to generate said virtual image by mapping a texture on saidplate-shaped polygon and shooting the polygon with said virtual camera.8. The non-transitory storage medium according to claim 1, wherein saidinformation processing program further comprises: position calculationinstructions for calculating positional relation between said imagepick-up apparatus and said search target based on the image of saidsubject; and virtual camera arrangement instructions for arranging saidvirtual camera in accordance with said positional relation calculated inaccordance with said position calculation instructions.
 9. Thenon-transitory storage medium according to claim 3, wherein saidinformation processing program further comprises: first search targetposition detection instructions for detecting a position of the firstsearch target from the image of said subject; first search targetposition calculation instructions for calculating positional relationbetween said image pick-up apparatus and said first search target basedon the position detected in accordance with said first search targetposition detection instructions; and virtual camera arrangementinstructions for arranging said virtual camera in accordance with thepositional relation calculated in accordance with said first searchtarget position calculation instructions, wherein said virtual imagegeneration instructions are adapted to generate said virtual image byshooting with said virtual camera, at least said second event thatoccurs in accordance with said event occurrence instructions.
 10. Aninformation processing apparatus connected to an image pick-up apparatusand a display, comprising: an image obtaining unit for obtaining animage of a subject picked up by said image pick-up apparatus; a searchtarget detection unit for detecting a search target from the image ofsaid subject obtained by said image obtaining unit; an event occurrenceunit for causing an event to occur in a virtual space in accordance withvariation in number of said search targets that can be detected by saidsearch target detection unit; a virtual image generation unit forgenerating a virtual image by shooting said event that occurs by meansof said event occurrence unit with a virtual camera arranged in saidvirtual space; and a display control unit for causing said display todisplay said virtual image such that it is visually recognized by a useras superimposed on the image of said subject or on said subject seenthrough a screen of said display.
 11. An information processing methodprocessed by a control unit of an information processing apparatusconnected to an image pick-up apparatus and a display, comprising thesteps of: obtaining an image of a subject picked up by said imagepick-up apparatus; detecting a search target from the obtained image ofsaid subject; causing an event to occur in a virtual space in accordancewith variation in number of said search targets that can be detected;generating a virtual image by shooting said event that occurs with avirtual camera arranged in said virtual space; and causing said displayto display said virtual image such that it is visually recognized by auser as superimposed on the image of said subject or on said subjectseen through a screen of said display.
 12. An information processingsystem, comprising: a search target arranged in a subject; an imagepick-up apparatus for picking up an image of said subject; aninformation processing apparatus connected to said image pick-upapparatus; and a display connected to said information processingapparatus, said information processing apparatus including an imageobtaining unit for obtaining the image of the subject picked up by saidimage pick-up apparatus, a search target detection unit for detectingsaid search target from the image of said subject obtained by said imageobtaining unit, an event occurrence unit for causing an event to occurin a virtual space in accordance with variation in number of said searchtargets that can be detected by said search target detection unit, avirtual image generation unit for generating a virtual image by shootingsaid event that occurs by means of said event occurrence unit with avirtual camera arranged in said virtual space, and a display controlunit for causing said display to display said virtual image such that itis visually recognized by a user as superimposed on the image of saidsubject or on said subject seen through a screen of said display.